{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Custom Containers with Vertex AI Training\n",
    "\n",
    "## Learning objectives\n",
    "1. Learn how to create a train and a validation split with BigQuery.\n",
    "1. Learn how to wrap a machine learning model into a Docker container and train in on Vertex AI.\n",
    "1. Learn how to use the hyperparameter tuning engine on Vertex AI to find the best hyperparameters.\n",
    "1. Learn how to deploy a trained machine learning model on Vertex AI as a REST API and query it.\n",
    "\n",
    "In this lab, you develop, package as a docker image, and run on **Vertex AI Training** a training application that trains a multi-class classification model that predicts the type of forest cover from cartographic data. The [dataset](../../../datasets/covertype/README.md) used in the lab is based on **Covertype Data Set** from UCI Machine Learning Repository.\n",
    "\n",
    "The training code uses `scikit-learn` for data pre-processing and modeling. The code has been instrumented using the `hypertune` package so it can be used with **Vertex AI** hyperparameter tuning.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__  in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/lab-01_vertex.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "google-cloud-aiplatform==1.18.2\n"
     ]
    }
   ],
   "source": [
    "!pip freeze | grep google-cloud-aiplatform || pip install google-cloud-aiplatform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Restart the kernel by clicking **Kernel > Restart Kernel > Restart**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud import bigquery\n",
    "import pandas as pd\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure environment settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set location paths, connections strings, and other environment settings. Make sure to update   `REGION`, and `ARTIFACT_STORE`  with the settings reflecting your lab environment. \n",
    "\n",
    "- `REGION` - the compute region for Vertex AI Training and Prediction\n",
    "- `ARTIFACT_STORE` - A GCS bucket in the created in the same region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "\n",
    "PROJECT_ID = !(gcloud config get-value core/project)\n",
    "PROJECT_ID = PROJECT_ID[0]\n",
    "\n",
    "ARTIFACT_STORE = f'gs://{PROJECT_ID}-vertex'\n",
    "\n",
    "DATA_ROOT = f'{ARTIFACT_STORE}/data'\n",
    "JOB_DIR_ROOT = f'{ARTIFACT_STORE}/jobs'\n",
    "TRAINING_FILE_PATH = f'{DATA_ROOT}/training/dataset.csv'\n",
    "VALIDATION_FILE_PATH = f'{DATA_ROOT}/validation/dataset.csv'\n",
    "API_ENDPOINT = f'{REGION}-aiplatform.googleapis.com'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['JOB_DIR_ROOT'] = JOB_DIR_ROOT\n",
    "os.environ['TRAINING_FILE_PATH'] = TRAINING_FILE_PATH\n",
    "os.environ['VALIDATION_FILE_PATH'] = VALIDATION_FILE_PATH\n",
    "os.environ['PROJECT_ID'] = PROJECT_ID\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the `ARTIFACT_STORE` bucket if it's not there. Note that this bucket should be created in the region specified in the variable `REGION` (if you have already a bucket with this name in a different region than `REGION`, you may want to change the `ARTIFACT_STORE` name so that you can recreate a bucket in `REGION` with the command in the cell below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://qwiklabs-gcp-00-a557e69ca320-vertex/...\n"
     ]
    }
   ],
   "source": [
    "!gsutil ls | grep ^{ARTIFACT_STORE}/$ || gsutil mb -l {REGION} {ARTIFACT_STORE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the dataset into BigQuery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset 'qwiklabs-gcp-00-a557e69ca320:covertype_dataset' successfully created.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Waiting on bqjob_r584d2e36a6382e5c_00000184e1f10a06_1 ... (2s) Current status: DONE   \n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "DATASET_LOCATION=US\n",
    "DATASET_ID=covertype_dataset\n",
    "TABLE_ID=covertype\n",
    "DATA_SOURCE=gs://cloud-training/OCBL203/workshop-datasets/dataset.csv\n",
    "SCHEMA=Elevation:INTEGER,\\\n",
    "Aspect:INTEGER,\\\n",
    "Slope:INTEGER,\\\n",
    "Horizontal_Distance_To_Hydrology:INTEGER,\\\n",
    "Vertical_Distance_To_Hydrology:INTEGER,\\\n",
    "Horizontal_Distance_To_Roadways:INTEGER,\\\n",
    "Hillshade_9am:INTEGER,\\\n",
    "Hillshade_Noon:INTEGER,\\\n",
    "Hillshade_3pm:INTEGER,\\\n",
    "Horizontal_Distance_To_Fire_Points:INTEGER,\\\n",
    "Wilderness_Area:STRING,\\\n",
    "Soil_Type:STRING,\\\n",
    "Cover_Type:INTEGER\n",
    "\n",
    "bq --location=$DATASET_LOCATION --project_id=$PROJECT_ID mk --dataset $DATASET_ID\n",
    "\n",
    "bq --project_id=$PROJECT_ID --dataset_id=$DATASET_ID load \\\n",
    "--source_format=CSV \\\n",
    "--skip_leading_rows=1 \\\n",
    "--replace \\\n",
    "$TABLE_ID \\\n",
    "$DATA_SOURCE \\\n",
    "$SCHEMA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Covertype dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query complete after 0.00s: 100%|██████████| 2/2 [00:00<00:00, 1028.39query/s]                        \n",
      "Downloading: 100%|██████████| 100000/100000 [00:01<00:00, 88740.79rows/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Elevation</th>\n",
       "      <th>Aspect</th>\n",
       "      <th>Slope</th>\n",
       "      <th>Horizontal_Distance_To_Hydrology</th>\n",
       "      <th>Vertical_Distance_To_Hydrology</th>\n",
       "      <th>Horizontal_Distance_To_Roadways</th>\n",
       "      <th>Hillshade_9am</th>\n",
       "      <th>Hillshade_Noon</th>\n",
       "      <th>Hillshade_3pm</th>\n",
       "      <th>Horizontal_Distance_To_Fire_Points</th>\n",
       "      <th>Wilderness_Area</th>\n",
       "      <th>Soil_Type</th>\n",
       "      <th>Cover_Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2085</td>\n",
       "      <td>256</td>\n",
       "      <td>18</td>\n",
       "      <td>150</td>\n",
       "      <td>27</td>\n",
       "      <td>738</td>\n",
       "      <td>176</td>\n",
       "      <td>248</td>\n",
       "      <td>208</td>\n",
       "      <td>914</td>\n",
       "      <td>Cache</td>\n",
       "      <td>C2702</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2125</td>\n",
       "      <td>256</td>\n",
       "      <td>20</td>\n",
       "      <td>30</td>\n",
       "      <td>12</td>\n",
       "      <td>871</td>\n",
       "      <td>169</td>\n",
       "      <td>248</td>\n",
       "      <td>215</td>\n",
       "      <td>300</td>\n",
       "      <td>Cache</td>\n",
       "      <td>C2702</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2146</td>\n",
       "      <td>256</td>\n",
       "      <td>34</td>\n",
       "      <td>150</td>\n",
       "      <td>62</td>\n",
       "      <td>1253</td>\n",
       "      <td>122</td>\n",
       "      <td>237</td>\n",
       "      <td>239</td>\n",
       "      <td>511</td>\n",
       "      <td>Cache</td>\n",
       "      <td>C2702</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2186</td>\n",
       "      <td>256</td>\n",
       "      <td>38</td>\n",
       "      <td>210</td>\n",
       "      <td>102</td>\n",
       "      <td>1294</td>\n",
       "      <td>109</td>\n",
       "      <td>232</td>\n",
       "      <td>244</td>\n",
       "      <td>552</td>\n",
       "      <td>Cache</td>\n",
       "      <td>C2702</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2831</td>\n",
       "      <td>256</td>\n",
       "      <td>25</td>\n",
       "      <td>277</td>\n",
       "      <td>183</td>\n",
       "      <td>1706</td>\n",
       "      <td>153</td>\n",
       "      <td>246</td>\n",
       "      <td>225</td>\n",
       "      <td>1485</td>\n",
       "      <td>Commanche</td>\n",
       "      <td>C2705</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99995</th>\n",
       "      <td>3136</td>\n",
       "      <td>254</td>\n",
       "      <td>12</td>\n",
       "      <td>319</td>\n",
       "      <td>60</td>\n",
       "      <td>5734</td>\n",
       "      <td>193</td>\n",
       "      <td>248</td>\n",
       "      <td>193</td>\n",
       "      <td>2467</td>\n",
       "      <td>Rawah</td>\n",
       "      <td>C7746</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99996</th>\n",
       "      <td>3242</td>\n",
       "      <td>254</td>\n",
       "      <td>12</td>\n",
       "      <td>636</td>\n",
       "      <td>148</td>\n",
       "      <td>3551</td>\n",
       "      <td>193</td>\n",
       "      <td>248</td>\n",
       "      <td>193</td>\n",
       "      <td>2010</td>\n",
       "      <td>Commanche</td>\n",
       "      <td>C7757</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99997</th>\n",
       "      <td>2071</td>\n",
       "      <td>255</td>\n",
       "      <td>12</td>\n",
       "      <td>234</td>\n",
       "      <td>63</td>\n",
       "      <td>342</td>\n",
       "      <td>192</td>\n",
       "      <td>247</td>\n",
       "      <td>193</td>\n",
       "      <td>247</td>\n",
       "      <td>Cache</td>\n",
       "      <td>C2706</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99998</th>\n",
       "      <td>3248</td>\n",
       "      <td>255</td>\n",
       "      <td>12</td>\n",
       "      <td>730</td>\n",
       "      <td>113</td>\n",
       "      <td>725</td>\n",
       "      <td>192</td>\n",
       "      <td>247</td>\n",
       "      <td>193</td>\n",
       "      <td>2724</td>\n",
       "      <td>Commanche</td>\n",
       "      <td>C7756</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99999</th>\n",
       "      <td>3153</td>\n",
       "      <td>255</td>\n",
       "      <td>12</td>\n",
       "      <td>404</td>\n",
       "      <td>116</td>\n",
       "      <td>2139</td>\n",
       "      <td>192</td>\n",
       "      <td>247</td>\n",
       "      <td>193</td>\n",
       "      <td>994</td>\n",
       "      <td>Commanche</td>\n",
       "      <td>C7756</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100000 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Elevation  Aspect  Slope  Horizontal_Distance_To_Hydrology  \\\n",
       "0           2085     256     18                               150   \n",
       "1           2125     256     20                                30   \n",
       "2           2146     256     34                               150   \n",
       "3           2186     256     38                               210   \n",
       "4           2831     256     25                               277   \n",
       "...          ...     ...    ...                               ...   \n",
       "99995       3136     254     12                               319   \n",
       "99996       3242     254     12                               636   \n",
       "99997       2071     255     12                               234   \n",
       "99998       3248     255     12                               730   \n",
       "99999       3153     255     12                               404   \n",
       "\n",
       "       Vertical_Distance_To_Hydrology  Horizontal_Distance_To_Roadways  \\\n",
       "0                                  27                              738   \n",
       "1                                  12                              871   \n",
       "2                                  62                             1253   \n",
       "3                                 102                             1294   \n",
       "4                                 183                             1706   \n",
       "...                               ...                              ...   \n",
       "99995                              60                             5734   \n",
       "99996                             148                             3551   \n",
       "99997                              63                              342   \n",
       "99998                             113                              725   \n",
       "99999                             116                             2139   \n",
       "\n",
       "       Hillshade_9am  Hillshade_Noon  Hillshade_3pm  \\\n",
       "0                176             248            208   \n",
       "1                169             248            215   \n",
       "2                122             237            239   \n",
       "3                109             232            244   \n",
       "4                153             246            225   \n",
       "...              ...             ...            ...   \n",
       "99995            193             248            193   \n",
       "99996            193             248            193   \n",
       "99997            192             247            193   \n",
       "99998            192             247            193   \n",
       "99999            192             247            193   \n",
       "\n",
       "       Horizontal_Distance_To_Fire_Points Wilderness_Area Soil_Type  \\\n",
       "0                                     914           Cache     C2702   \n",
       "1                                     300           Cache     C2702   \n",
       "2                                     511           Cache     C2702   \n",
       "3                                     552           Cache     C2702   \n",
       "4                                    1485       Commanche     C2705   \n",
       "...                                   ...             ...       ...   \n",
       "99995                                2467           Rawah     C7746   \n",
       "99996                                2010       Commanche     C7757   \n",
       "99997                                 247           Cache     C2706   \n",
       "99998                                2724       Commanche     C7756   \n",
       "99999                                 994       Commanche     C7756   \n",
       "\n",
       "       Cover_Type  \n",
       "0               5  \n",
       "1               2  \n",
       "2               2  \n",
       "3               2  \n",
       "4               1  \n",
       "...           ...  \n",
       "99995           1  \n",
       "99996           0  \n",
       "99997           2  \n",
       "99998           1  \n",
       "99999           1  \n",
       "\n",
       "[100000 rows x 13 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "SELECT *\n",
    "FROM `covertype_dataset.covertype`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create training and validation splits\n",
    "\n",
    "Use BigQuery to sample training and validation splits and save them to GCS storage\n",
    "### Create a training split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting on bqjob_r211d88d4e3252af9_00000184e1f205ba_1 ... (1s) Current status: DONE   \n"
     ]
    }
   ],
   "source": [
    "!bq query \\\n",
    "-n 0 \\\n",
    "--destination_table covertype_dataset.training \\\n",
    "--replace \\\n",
    "--use_legacy_sql=false \\\n",
    "'SELECT * \\\n",
    "FROM `covertype_dataset.covertype` AS cover \\\n",
    "WHERE \\\n",
    "MOD(ABS(FARM_FINGERPRINT(TO_JSON_STRING(cover))), 10) IN (1, 2, 3, 4)' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting on bqjob_r136482b6eefe5f5e_00000184e1f22d6e_1 ... (0s) Current status: DONE   \n"
     ]
    }
   ],
   "source": [
    "!bq extract \\\n",
    "--destination_format CSV \\\n",
    "covertype_dataset.training \\\n",
    "$TRAINING_FILE_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a validation split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Waiting on bqjob_r264815c23427384f_00000184e1f26c93_1 ... (1s) Current status: DONE   \n"
      ]
     }
   ],
   "source": [
    "# TODO 1: Your code goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Waiting on bqjob_r549ab8467daae02f_00000184e1f27e9e_1 ... (0s) Current status: DONE   \n"
      ]
     }
   ],
   "source": [
    "# Export the validation table to GCS\n",
    "# TODO 2: Your code goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(40009, 13)\n",
      "(9836, 13)\n"
     ]
    }
   ],
   "source": [
    "df_train = pd.read_csv(TRAINING_FILE_PATH)\n",
    "df_validation = pd.read_csv(VALIDATION_FILE_PATH)\n",
    "print(df_train.shape)\n",
    "print(df_validation.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Develop a training application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the `sklearn` training pipeline.\n",
    "\n",
    "The training pipeline preprocesses data by standardizing all numeric features using `sklearn.preprocessing.StandardScaler` and encoding all categorical features using `sklearn.preprocessing.OneHotEncoder`. It uses stochastic gradient descent linear classifier (`SGDClassifier`) for modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feature_indexes = slice(0, 10)\n",
    "categorical_feature_indexes = slice(10, 12)\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', StandardScaler(), numeric_feature_indexes),\n",
    "        ('cat', OneHotEncoder(), categorical_feature_indexes) \n",
    "    ])\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('classifier', SGDClassifier(loss='log', tol=1e-3))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert all numeric features to `float64`\n",
    "\n",
    "To avoid warning messages from `StandardScaler` all numeric features are converted to `float64`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_features_type_map = {feature: 'float64' for feature in df_train.columns[numeric_feature_indexes]}\n",
    "\n",
    "df_train = df_train.astype(num_features_type_map)\n",
    "df_validation = df_validation.astype(num_features_type_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the pipeline locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
     {
      "data": {
       "text/plain": [
        "Pipeline(steps=[('preprocessor',\n",
        "                 ColumnTransformer(transformers=[('num', StandardScaler(),\n",
        "                                                  slice(0, 10, None)),\n",
        "                                                 ('cat', OneHotEncoder(),\n",
        "                                                  slice(10, 12, None))])),\n",
        "                ('classifier',\n",
        "                 SGDClassifier(alpha=0.001, loss='log', max_iter=200))])"
       ]
      },
      "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
   ],
   "source": [
    "X_train = df_train.drop('Cover_Type', axis=1)\n",
    "y_train = df_train['Cover_Type']\n",
    "X_validation = df_validation.drop('Cover_Type', axis=1)\n",
    "y_validation = df_validation['Cover_Type']\n",
    "\n",
    "pipeline.set_params(classifier__alpha=0.001, classifier__max_iter=200)\n",
    "pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate the trained model's accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "0.6959129727531517\n"
      ]
     }
   ],
   "source": [
    "accuracy = pipeline.score(X_validation, y_validation)\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the hyperparameter tuning application.\n",
    "Since the training run on this dataset is computationally expensive you can benefit from running a distributed hyperparameter tuning job on Vertex AI Training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_APP_FOLDER = 'training_app'\n",
    "os.makedirs(TRAINING_APP_FOLDER, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write the tuning script. \n",
    "\n",
    "Notice the use of the `hypertune` package to report the `accuracy` optimization metric to Vertex AI hyperparameter tuning service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Writing training_app/train.py\n"
      ]
     }
   ],
   "source": [
    "%%writefile {TRAINING_APP_FOLDER}/train.py\n",
    "import os\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "import fire\n",
    "import hypertune\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "\n",
    "def train_evaluate(job_dir, training_dataset_path, validation_dataset_path, alpha, max_iter, hptune):\n",
    "    \n",
    "    df_train = pd.read_csv(training_dataset_path)\n",
    "    df_validation = pd.read_csv(validation_dataset_path)\n",
    "\n",
    "    if not hptune:\n",
    "        df_train = pd.concat([df_train, df_validation])\n",
    "\n",
    "    numeric_feature_indexes = slice(0, 10)\n",
    "    categorical_feature_indexes = slice(10, 12)\n",
    "\n",
    "    preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', StandardScaler(), numeric_feature_indexes),\n",
    "        ('cat', OneHotEncoder(), categorical_feature_indexes) \n",
    "    ])\n",
    "\n",
    "    pipeline = Pipeline([\n",
    "        ('preprocessor', preprocessor),\n",
    "        ('classifier', SGDClassifier(loss='log',tol=1e-3))\n",
    "    ])\n",
    "\n",
    "    num_features_type_map = {feature: 'float64' for feature in df_train.columns[numeric_feature_indexes]}\n",
    "    df_train = df_train.astype(num_features_type_map)\n",
    "    df_validation = df_validation.astype(num_features_type_map) \n",
    "\n",
    "    print('Starting training: alpha={}, max_iter={}'.format(alpha, max_iter))\n",
    "    X_train = df_train.drop('Cover_Type', axis=1)\n",
    "    y_train = df_train['Cover_Type']\n",
    "\n",
    "    pipeline.set_params(classifier__alpha=alpha, classifier__max_iter=max_iter)\n",
    "    pipeline.fit(X_train, y_train)\n",
    "\n",
    "    if hptune:\n",
    "        X_validation = df_validation.drop('Cover_Type', axis=1)\n",
    "        y_validation = df_validation['Cover_Type']\n",
    "        accuracy = pipeline.score(X_validation, y_validation)\n",
    "        print('Model accuracy: {}'.format(accuracy))\n",
    "        # Log it with hypertune\n",
    "        hpt = hypertune.HyperTune()\n",
    "        hpt.report_hyperparameter_tuning_metric(\n",
    "          hyperparameter_metric_tag='accuracy',\n",
    "          metric_value=accuracy\n",
    "        )\n",
    "\n",
    "    # Save the model\n",
    "    if not hptune:\n",
    "        model_filename = 'model.pkl'\n",
    "        with open(model_filename, 'wb') as model_file:\n",
    "            pickle.dump(pipeline, model_file)\n",
    "        gcs_model_path = \"{}/{}\".format(job_dir, model_filename)\n",
    "        subprocess.check_call(['gsutil', 'cp', model_filename, gcs_model_path], stderr=sys.stdout)\n",
    "        print(\"Saved model in: {}\".format(gcs_model_path)) \n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "    fire.Fire(train_evaluate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Package the script into a docker image.\n",
    "\n",
    "Notice that we are installing specific versions of `scikit-learn` and `pandas` in the training image. This is done to make sure that the training runtime in the training container is aligned with the serving runtime in the serving container. \n",
    "\n",
    "Make sure to update the URI for the base image so that it points to your project's **Container Registry**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Complete the Dockerfile below so that it copies the 'train.py' file into the container\n",
    "at `/app` and runs it when the container is started. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Writing training_app/Dockerfile\n"
      ]
     }
   ],
   "source": [
    "%%writefile {TRAINING_APP_FOLDER}/Dockerfile\n",
    "\n",
    "FROM gcr.io/deeplearning-platform-release/base-cpu\n",
    "RUN pip install -U fire cloudml-hypertune scikit-learn==0.20.4 pandas==0.24.2\n",
    "\n",
    "# TODO 3: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the docker image. \n",
    "\n",
    "You use **Cloud Build** to build the image and push it your project's **Container Registry**. As you use the remote cloud service to build the image, you don't need a local installation of Docker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_NAME='trainer_image'\n",
    "IMAGE_TAG='latest'\n",
    "IMAGE_URI='gcr.io/{}/{}:{}'.format(PROJECT_ID, IMAGE_NAME, IMAGE_TAG)\n",
    "\n",
    "os.environ['IMAGE_URI'] = IMAGE_URI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Creating temporary tarball archive of 2 file(s) totalling 2.6 KiB before compression.\n",
       "Uploading tarball of [training_app] to [gs://qwiklabs-gcp-00-a557e69ca320_cloudbuild/source/1670238081.394112-bf0698630eb24898bb78d8d6e06e60e8.tgz]\n",
       "Created [https://cloudbuild.googleapis.com/v1/projects/qwiklabs-gcp-00-a557e69ca320/locations/global/builds/345c769e-d4c9-4deb-a2af-374189bf9d46].\n",
       "Logs are available at [ https://console.cloud.google.com/cloud-build/builds/345c769e-d4c9-4deb-a2af-374189bf9d46?project=676946291090 ].\n",
       "----------------------------- REMOTE BUILD OUTPUT ------------------------------\n",
       "starting build \"345c769e-d4c9-4deb-a2af-374189bf9d46\"\n",
       "\n",
       "FETCHSOURCE\n",
       "Fetching storage object: gs://qwiklabs-gcp-00-a557e69ca320_cloudbuild/source/1670238081.394112-bf0698630eb24898bb78d8d6e06e60e8.tgz#1670238082606717\n",
       "Copying gs://qwiklabs-gcp-00-a557e69ca320_cloudbuild/source/1670238081.394112-bf0698630eb24898bb78d8d6e06e60e8.tgz#1670238082606717...\n",
       "/ [1 files][  1.2 KiB/  1.2 KiB]                                                \n",
       "Operation completed over 1 objects/1.2 KiB.\n",
       "BUILD\n",
       "Already have image (with digest): gcr.io/cloud-builders/docker\n",
       "Sending build context to Docker daemon   5.12kB\n",
       "Step 1/5 : FROM gcr.io/deeplearning-platform-release/base-cpu\n",
       "latest: Pulling from deeplearning-platform-release/base-cpu\n",
       "eaead16dc43b: Pulling fs layer\n",
       "6529fe2a781d: Pulling fs layer\n",
       "470effa3df00: Pulling fs layer\n",
       "4f4fb700ef54: Pulling fs layer\n",
       "81f7870f3a99: Pulling fs layer\n",
       "97e572c9d0d3: Pulling fs layer\n",
       "67c86225922e: Pulling fs layer\n",
       "f0be7e3b4260: Pulling fs layer\n",
       "27519836c4fe: Pulling fs layer\n",
       "8114d85b6dca: Pulling fs layer\n",
       "4f2efc2c2115: Pulling fs layer\n",
       "8b38fa015e72: Pulling fs layer\n",
       "15b21dd7632f: Pulling fs layer\n",
       "15c1e55a1fdc: Pulling fs layer\n",
       "92394077f3fc: Pulling fs layer\n",
       "adc7c41f6a21: Pulling fs layer\n",
       "dd2f3ccdd187: Pulling fs layer\n",
       "7d108414db7f: Pulling fs layer\n",
       "c0c5eb2fe5ca: Pulling fs layer\n",
       "87f6b5e0c563: Pulling fs layer\n",
       "9993b2c47e68: Pulling fs layer\n",
       "52e0ea262be3: Pulling fs layer\n",
       "8b38fa015e72: Waiting\n",
       "15b21dd7632f: Waiting\n",
       "15c1e55a1fdc: Waiting\n",
       "92394077f3fc: Waiting\n",
       "adc7c41f6a21: Waiting\n",
       "dd2f3ccdd187: Waiting\n",
       "4f4fb700ef54: Waiting\n",
       "7d108414db7f: Waiting\n",
       "81f7870f3a99: Waiting\n",
       "97e572c9d0d3: Waiting\n",
       "c0c5eb2fe5ca: Waiting\n",
       "87f6b5e0c563: Waiting\n",
       "67c86225922e: Waiting\n",
       "f0be7e3b4260: Waiting\n",
       "9993b2c47e68: Waiting\n",
       "52e0ea262be3: Waiting\n",
       "27519836c4fe: Waiting\n",
       "4f2efc2c2115: Waiting\n",
       "8114d85b6dca: Waiting\n",
       "470effa3df00: Verifying Checksum\n",
       "470effa3df00: Download complete\n",
       "4f4fb700ef54: Verifying Checksum\n",
       "4f4fb700ef54: Download complete\n",
       "6529fe2a781d: Verifying Checksum\n",
       "6529fe2a781d: Download complete\n",
       "eaead16dc43b: Verifying Checksum\n",
       "eaead16dc43b: Download complete\n",
       "67c86225922e: Verifying Checksum\n",
       "67c86225922e: Download complete\n",
       "f0be7e3b4260: Verifying Checksum\n",
       "f0be7e3b4260: Download complete\n",
       "27519836c4fe: Verifying Checksum\n",
       "27519836c4fe: Download complete\n",
       "97e572c9d0d3: Verifying Checksum\n",
       "97e572c9d0d3: Download complete\n",
       "4f2efc2c2115: Verifying Checksum\n",
       "4f2efc2c2115: Download complete\n",
       "8b38fa015e72: Verifying Checksum\n",
       "8b38fa015e72: Download complete\n",
       "15b21dd7632f: Verifying Checksum\n",
       "15b21dd7632f: Download complete\n",
       "15c1e55a1fdc: Verifying Checksum\n",
       "15c1e55a1fdc: Download complete\n",
       "92394077f3fc: Verifying Checksum\n",
       "92394077f3fc: Download complete\n",
       "8114d85b6dca: Verifying Checksum\n",
       "8114d85b6dca: Download complete\n",
       "81f7870f3a99: Verifying Checksum\n",
       "81f7870f3a99: Download complete\n",
       "adc7c41f6a21: Verifying Checksum\n",
       "adc7c41f6a21: Download complete\n",
       "dd2f3ccdd187: Verifying Checksum\n",
       "dd2f3ccdd187: Download complete\n",
       "c0c5eb2fe5ca: Verifying Checksum\n",
       "c0c5eb2fe5ca: Download complete\n",
       "7d108414db7f: Verifying Checksum\n",
       "7d108414db7f: Download complete\n",
       "87f6b5e0c563: Verifying Checksum\n",
       "87f6b5e0c563: Download complete\n",
       "52e0ea262be3: Download complete\n",
       "eaead16dc43b: Pull complete\n",
       "6529fe2a781d: Pull complete\n",
       "470effa3df00: Pull complete\n",
       "4f4fb700ef54: Pull complete\n",
       "9993b2c47e68: Verifying Checksum\n",
       "9993b2c47e68: Download complete\n",
       "81f7870f3a99: Pull complete\n",
       "97e572c9d0d3: Pull complete\n",
       "67c86225922e: Pull complete\n",
       "f0be7e3b4260: Pull complete\n",
       "27519836c4fe: Pull complete\n",
       "8114d85b6dca: Pull complete\n",
       "4f2efc2c2115: Pull complete\n",
       "8b38fa015e72: Pull complete\n",
       "15b21dd7632f: Pull complete\n",
       "15c1e55a1fdc: Pull complete\n",
       "92394077f3fc: Pull complete\n",
       "adc7c41f6a21: Pull complete\n",
       "dd2f3ccdd187: Pull complete\n",
       "7d108414db7f: Pull complete\n",
       "c0c5eb2fe5ca: Pull complete\n",
       "87f6b5e0c563: Pull complete\n",
       "9993b2c47e68: Pull complete\n",
       "52e0ea262be3: Pull complete\n",
       "Digest: sha256:d2a00748eaa993fdb5fc941ee6695e6142b53192bff99e74b76e85a876f52379\n",
       "Status: Downloaded newer image for gcr.io/deeplearning-platform-release/base-cpu:latest\n",
       " ---> f5c89f738019\n",
       "Step 2/5 : RUN pip install -U fire cloudml-hypertune scikit-learn==0.20.4 pandas==0.24.2\n",
       " ---> Running in 94a14ddbcb90\n",
       "Collecting fire\n",
       "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
       "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.7/87.7 kB 4.0 MB/s eta 0:00:00\n",
       "  Preparing metadata (setup.py): started\n",
       "  Preparing metadata (setup.py): finished with status 'done'\n",
       "Collecting cloudml-hypertune\n",
       "  Downloading cloudml-hypertune-0.1.0.dev6.tar.gz (3.2 kB)\n",
       "  Preparing metadata (setup.py): started\n",
       "  Preparing metadata (setup.py): finished with status 'done'\n",
       "Collecting scikit-learn==0.20.4\n",
       "  Downloading scikit_learn-0.20.4-cp37-cp37m-manylinux1_x86_64.whl (5.4 MB)\n",
       "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 41.1 MB/s eta 0:00:00\n",
       "Collecting pandas==0.24.2\n",
       "  Downloading pandas-0.24.2-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)\n",
       "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 MB 60.9 MB/s eta 0:00:00\n",
       "Requirement already satisfied: scipy>=0.13.3 in /opt/conda/lib/python3.7/site-packages (from scikit-learn==0.20.4) (1.7.3)\n",
       "Requirement already satisfied: numpy>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from scikit-learn==0.20.4) (1.21.6)\n",
       "Requirement already satisfied: pytz>=2011k in /opt/conda/lib/python3.7/site-packages (from pandas==0.24.2) (2022.5)\n",
       "Requirement already satisfied: python-dateutil>=2.5.0 in /opt/conda/lib/python3.7/site-packages (from pandas==0.24.2) (2.8.2)\n",
       "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from fire) (1.16.0)\n",
       "Collecting termcolor\n",
       "  Downloading termcolor-2.1.1-py3-none-any.whl (6.2 kB)\n",
       "Building wheels for collected packages: fire, cloudml-hypertune\n",
       "  Building wheel for fire (setup.py): started\n",
       "  Building wheel for fire (setup.py): finished with status 'done'\n",
       "  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115942 sha256=2621a6ffa9f82bebb849a404759d41559e7dbf977bac6d03688f8d88fb8f987e\n",
       "  Stored in directory: /root/.cache/pip/wheels/83/21/65/2ac62db55efa6e6edfad09f4e315aa82a35ab138f51e784fb1\n",
       "  Building wheel for cloudml-hypertune (setup.py): started\n",
       "  Building wheel for cloudml-hypertune (setup.py): finished with status 'done'\n",
       "  Created wheel for cloudml-hypertune: filename=cloudml_hypertune-0.1.0.dev6-py2.py3-none-any.whl size=3987 sha256=b3007be8aa09637e1579c4f0d422a4994124f850a865cb967e238c8689ab9126\n",
       "  Stored in directory: /root/.cache/pip/wheels/7c/fb/ed/cfc98e70373dfe12db85fffab293e3153162f63de2f6aa5473\n",
       "Successfully built fire cloudml-hypertune\n",
       "Installing collected packages: cloudml-hypertune, termcolor, scikit-learn, pandas, fire\n",
       "  Attempting uninstall: scikit-learn\n",
       "    Found existing installation: scikit-learn 1.0.2\n",
       "    Uninstalling scikit-learn-1.0.2:\n",
       "      Successfully uninstalled scikit-learn-1.0.2\n",
       "  Attempting uninstall: pandas\n",
       "    Found existing installation: pandas 1.3.5\n",
       "    Uninstalling pandas-1.3.5:\n",
       "      Successfully uninstalled pandas-1.3.5\n",
       "\u001b[91mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
       "visions 0.7.5 requires pandas>=0.25.3, but you have pandas 0.24.2 which is incompatible.\n",
       "statsmodels 0.13.2 requires pandas>=0.25, but you have pandas 0.24.2 which is incompatible.\n",
       "seaborn 0.12.1 requires pandas>=0.25, but you have pandas 0.24.2 which is incompatible.\n",
       "phik 0.12.2 requires pandas>=0.25.1, but you have pandas 0.24.2 which is incompatible.\n",
       "pandas-profiling 3.4.0 requires pandas!=1.4.0,<1.6,>1.1, but you have pandas 0.24.2 which is incompatible.\n",
       "\u001b[0mSuccessfully installed cloudml-hypertune-0.1.0.dev6 fire-0.4.0 pandas-0.24.2 scikit-learn-0.20.4 termcolor-2.1.1\n",
       "\u001b[91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
       "\u001b[0mRemoving intermediate container 94a14ddbcb90\n",
       " ---> 0bdc6aee5dab\n",
       "Step 3/5 : WORKDIR /app\n",
       " ---> Running in b2dd25f6a833\n",
       "Removing intermediate container b2dd25f6a833\n",
       " ---> cc39185a44a8\n",
       "Step 4/5 : COPY train.py .\n",
       " ---> e2a6e69d6048\n",
       "Step 5/5 : ENTRYPOINT [\"python\", \"train.py\"]\n",
       " ---> Running in 0a8301d31dd9\n",
       "Removing intermediate container 0a8301d31dd9\n",
       " ---> 3da598a4552d\n",
       "Successfully built 3da598a4552d\n",
       "Successfully tagged gcr.io/qwiklabs-gcp-00-a557e69ca320/trainer_image:latest\n",
       "PUSH\n",
       "Pushing gcr.io/qwiklabs-gcp-00-a557e69ca320/trainer_image:latest\n",
       "The push refers to repository [gcr.io/qwiklabs-gcp-00-a557e69ca320/trainer_image]\n",
       "8afbf796e19f: Preparing\n",
       "329804c2665d: Preparing\n",
       "a20a316c8d10: Preparing\n",
       "2ce63e75bec6: Preparing\n",
       "ee470c1be53d: Preparing\n",
       "affd8f958786: Preparing\n",
       "d9807b9e6598: Preparing\n",
       "159d73d7356c: Preparing\n",
       "b6a3d584445c: Preparing\n",
       "9c2c2f9a22a6: Preparing\n",
       "11e4526fac3c: Preparing\n",
       "60f7ab6b2229: Preparing\n",
       "9b55c6d23a55: Preparing\n",
       "bb078aa13c77: Preparing\n",
       "c99f4484fa81: Preparing\n",
       "14a72aa542a8: Preparing\n",
       "d207b04446b4: Preparing\n",
       "9e49809a77aa: Preparing\n",
       "9b3c3bd1f8de: Preparing\n",
       "5f70bf18a086: Preparing\n",
       "dbf08d497500: Preparing\n",
       "83b459ce565c: Preparing\n",
       "5f70bf18a086: Preparing\n",
       "3cab8b825015: Preparing\n",
       "b362faadeffc: Preparing\n",
       "f4462d5b2da2: Preparing\n",
       "affd8f958786: Waiting\n",
       "d9807b9e6598: Waiting\n",
       "159d73d7356c: Waiting\n",
       "b6a3d584445c: Waiting\n",
       "9c2c2f9a22a6: Waiting\n",
       "11e4526fac3c: Waiting\n",
       "60f7ab6b2229: Waiting\n",
       "9b55c6d23a55: Waiting\n",
       "bb078aa13c77: Waiting\n",
       "c99f4484fa81: Waiting\n",
       "14a72aa542a8: Waiting\n",
       "d207b04446b4: Waiting\n",
       "9e49809a77aa: Waiting\n",
       "9b3c3bd1f8de: Waiting\n",
       "5f70bf18a086: Waiting\n",
       "dbf08d497500: Waiting\n",
       "83b459ce565c: Waiting\n",
       "3cab8b825015: Waiting\n",
       "b362faadeffc: Waiting\n",
       "f4462d5b2da2: Waiting\n",
       "2ce63e75bec6: Mounted from deeplearning-platform-release/base-cpu\n",
       "ee470c1be53d: Mounted from deeplearning-platform-release/base-cpu\n",
       "affd8f958786: Mounted from deeplearning-platform-release/base-cpu\n",
       "d9807b9e6598: Mounted from deeplearning-platform-release/base-cpu\n",
       "329804c2665d: Pushed\n",
       "8afbf796e19f: Pushed\n",
       "159d73d7356c: Mounted from deeplearning-platform-release/base-cpu\n",
       "b6a3d584445c: Mounted from deeplearning-platform-release/base-cpu\n",
       "9c2c2f9a22a6: Mounted from deeplearning-platform-release/base-cpu\n",
       "11e4526fac3c: Mounted from deeplearning-platform-release/base-cpu\n",
       "60f7ab6b2229: Mounted from deeplearning-platform-release/base-cpu\n",
       "9b55c6d23a55: Mounted from deeplearning-platform-release/base-cpu\n",
       "bb078aa13c77: Mounted from deeplearning-platform-release/base-cpu\n",
       "14a72aa542a8: Mounted from deeplearning-platform-release/base-cpu\n",
       "c99f4484fa81: Mounted from deeplearning-platform-release/base-cpu\n",
       "d207b04446b4: Mounted from deeplearning-platform-release/base-cpu\n",
       "9e49809a77aa: Mounted from deeplearning-platform-release/base-cpu\n",
       "5f70bf18a086: Layer already exists\n",
       "9b3c3bd1f8de: Mounted from deeplearning-platform-release/base-cpu\n",
       "dbf08d497500: Mounted from deeplearning-platform-release/base-cpu\n",
       "3cab8b825015: Mounted from deeplearning-platform-release/base-cpu\n",
       "83b459ce565c: Mounted from deeplearning-platform-release/base-cpu\n",
       "f4462d5b2da2: Layer already exists\n",
       "b362faadeffc: Mounted from deeplearning-platform-release/base-cpu\n",
       "a20a316c8d10: Pushed\n",
       "latest: digest: sha256:aabfd1fe7a2905d9b75f81c9f3941fd2876b6b6a1b060ef642c223e6a5193628 size: 5754\n",
       "DONE\n",
       "--------------------------------------------------------------------------------\n",
       "ID                                    CREATE_TIME                DURATION  SOURCE                                                                                                      IMAGES                                                       STATUS\n",
       "345c769e-d4c9-4deb-a2af-374189bf9d46  2022-12-05T11:01:22+00:00  2M12S     gs://qwiklabs-gcp-00-a557e69ca320_cloudbuild/source/1670238081.394112-bf0698630eb24898bb78d8d6e06e60e8.tgz  gcr.io/qwiklabs-gcp-00-a557e69ca320/trainer_image (+1 more)  SUCCESS\n"
      ]
     }
   ],
   "source": [
    "!gcloud builds submit --tag $IMAGE_URI $TRAINING_APP_FOLDER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit an Vertex AI hyperparameter tuning job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the hyperparameter configuration file. \n",
    "Recall that the training code uses `SGDClassifier`. The training application has been designed to accept two hyperparameters that control `SGDClassifier`:\n",
    "- Max iterations\n",
    "- Alpha\n",
    "\n",
    "The file below configures Vertex AI hypertuning to run up to 5 trials in parallel and to choose from two discrete values of `max_iter` and the linear range between `1.0e-4` and `1.0e-1` for `alpha`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "TIMESTAMP = time.strftime(\"%Y%m%d_%H%M%S\")\n",
    "JOB_NAME = f\"forestcover_tuning_{TIMESTAMP}\"\n",
    "JOB_DIR = f\"{JOB_DIR_ROOT}/{JOB_NAME}\"\n",
    "\n",
    "os.environ['JOB_NAME'] = JOB_NAME\n",
    "os.environ['JOB_DIR'] = JOB_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Complete the `config.yaml` file generated below so that the hyperparameter\n",
    "tunning engine try for parameter values\n",
    "* `max_iter` the two values 10 and 20\n",
    "* `alpha` a linear range of values between  1.0e-4 and 1.0e-1\n",
    "\n",
    "Also complete the `gcloud` command to start the hyperparameter tuning job with a max trial count and\n",
    "a max number of parallel trials both of 5 each. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "JOB_NAME: forestcover_tuning_20221205_110340\n"
      ]
     },
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
       "Using endpoint [https://us-central1-aiplatform.googleapis.com/]\n",
       "Hyperparameter tuning job [492097561566380032] submitted successfully.\n",
       "\n",
       "Your job is still active. You may view the status of your job with the command\n",
       "\n",
       "  $ gcloud ai hp-tuning-jobs describe 492097561566380032 --region=us-central1\n",
       "\n",
       "Job State: JOB_STATE_PENDING\n"
      ]
     }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "MACHINE_TYPE=\"n1-standard-4\"\n",
    "REPLICA_COUNT=1\n",
    "CONFIG_YAML=config.yaml\n",
    "\n",
    "cat <<EOF > $CONFIG_YAML\n",
    "studySpec:\n",
    "  metrics:\n",
    "  - metricId: accuracy\n",
    "    goal: MAXIMIZE\n",
    "  parameters:\n",
    "    \n",
    "  # TODO 4: Your code goes here\n",
    "\n",
    "  algorithm: ALGORITHM_UNSPECIFIED # results in Bayesian optimization\n",
    "trialJobSpec:\n",
    "  workerPoolSpecs:  \n",
    "  - machineSpec:\n",
    "      machineType: $MACHINE_TYPE\n",
    "    replicaCount: $REPLICA_COUNT\n",
    "    containerSpec:\n",
    "      imageUri: $IMAGE_URI\n",
    "      args:\n",
    "      - --job_dir=$JOB_DIR\n",
    "      - --training_dataset_path=$TRAINING_FILE_PATH\n",
    "      - --validation_dataset_path=$VALIDATION_FILE_PATH\n",
    "      - --hptune\n",
    "EOF\n",
    "\n",
    "gcloud ai hp-tuning-jobs create \\\n",
    "    --region=# TODO \\\n",
    "    --display-name=# TODO \\\n",
    "    --config=# TODO \\\n",
    "    --max-trial-count=# TODO \\\n",
    "    --parallel-trial-count=# TODO\n",
    "\n",
    "echo \"JOB_NAME: $JOB_NAME\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to the Vertex AI Training dashboard and view the progression of the HP tuning job under \"Hyperparameter Tuning Jobs\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve HP-tuning results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the job completes you can review the results using GCP Console or programmatically using the following functions (note that this code supposes that the metrics that the hyperparameter tuning engine optimizes is maximized): "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Complete the body of the function below to retrieve the best trial from the `JOBNAME`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieve_best_trial_from_job_name(jobname):\n",
    "    \n",
    "    # TODO 5: Your code goes here\n",
    "    \n",
    "    return best_trial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll need to wait for the hyperparameter job to complete before being able to retrieve the best job by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_trial = retrieve_best_trial_from_job_name(JOB_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrain the model with the best hyperparameters\n",
    "\n",
    "You can now retrain the model using the best hyperparameters and using combined training and validation splits as a training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure and run the training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = best_trial.parameters[0].value\n",
    "max_iter = best_trial.parameters[1].value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Using endpoint [https://us-central1-aiplatform.googleapis.com/]\n",
       "CustomJob [projects/676946291090/locations/us-central1/customJobs/5301941963598069760] is submitted successfully.\n",
       "\n",
       "Your job is still active. You may view the status of your job with the command\n",
       "\n",
       "  $ gcloud ai custom-jobs describe projects/676946291090/locations/us-central1/customJobs/5301941963598069760\n",
       "\n",
       "or continue streaming the logs with the command\n",
       "\n",
       "  $ gcloud ai custom-jobs stream-logs projects/676946291090/locations/us-central1/customJobs/5301941963598069760\n",
       "The model will be exported at: gs://qwiklabs-gcp-00-a557e69ca320-vertex/jobs/JOB_VERTEX_20221205_111439\n"
      ]
     }
   ],
   "source": [
    "TIMESTAMP = time.strftime(\"%Y%m%d_%H%M%S\")\n",
    "JOB_NAME = f\"JOB_VERTEX_{TIMESTAMP}\"\n",
    "JOB_DIR = f\"{JOB_DIR_ROOT}/{JOB_NAME}\"\n",
    "\n",
    "MACHINE_TYPE=\"n1-standard-4\"\n",
    "REPLICA_COUNT=1\n",
    "\n",
    "WORKER_POOL_SPEC = f\"\"\"\\\n",
    "machine-type={MACHINE_TYPE},\\\n",
    "replica-count={REPLICA_COUNT},\\\n",
    "container-image-uri={IMAGE_URI}\\\n",
    "\"\"\"\n",
    "\n",
    "ARGS = f\"\"\"\\\n",
    "--job_dir={JOB_DIR},\\\n",
    "--training_dataset_path={TRAINING_FILE_PATH},\\\n",
    "--validation_dataset_path={VALIDATION_FILE_PATH},\\\n",
    "--alpha={alpha},\\\n",
    "--max_iter={max_iter},\\\n",
    "--nohptune\\\n",
    "\"\"\"\n",
    "\n",
    "!gcloud ai custom-jobs create \\\n",
    "  --region={REGION} \\\n",
    "  --display-name={JOB_NAME} \\\n",
    "  --worker-pool-spec={WORKER_POOL_SPEC} \\\n",
    "  --args={ARGS}\n",
    "\n",
    "\n",
    "print(\"The model will be exported at:\", JOB_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the training output\n",
    "\n",
    "The training script saved the trained model as the 'model.pkl' in the `JOB_DIR` folder on GCS.\n",
    "\n",
    "**Note:** We need to wait for job triggered by the cell above to complete before running the cells below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "gs://qwiklabs-gcp-00-a557e69ca320-vertex/jobs/JOB_VERTEX_20221205_111439/model.pkl\n"
      ]
     }
   ],
   "source": [
    "!gsutil ls $JOB_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the model to Vertex AI Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = 'forest_cover_classifier_2'\n",
    "SERVING_CONTAINER_IMAGE_URI = 'us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.0-20:latest'\n",
    "SERVING_MACHINE_TYPE = \"n1-standard-2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading the trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Upload the trained model using `aiplatform.Model.upload`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Creating Model\n",
       "Create Model backing LRO: projects/676946291090/locations/us-central1/models/810790869438300160/operations/6142403360470335488\n",
       "Model created. Resource name: projects/676946291090/locations/us-central1/models/810790869438300160@1\n",
       "To use this Model in another session:\n",
       "model = aiplatform.Model('projects/676946291090/locations/us-central1/models/810790869438300160@1')\n"
      ]
     }
   ],
   "source": [
    "uploaded_model = # TODO 6: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploying the uploaded model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Deploy the model using `uploaded_model`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Creating Endpoint\n",
       "Create Endpoint backing LRO: projects/676946291090/locations/us-central1/endpoints/5390144648939307008/operations/2773710839197204480\n",
       "Endpoint created. Resource name: projects/676946291090/locations/us-central1/endpoints/5390144648939307008\n",
       "To use this Endpoint in another session:\n",
       "endpoint = aiplatform.Endpoint('projects/676946291090/locations/us-central1/endpoints/5390144648939307008')\n",
       "Deploying model to Endpoint : projects/676946291090/locations/us-central1/endpoints/5390144648939307008\n",
       "Deploy Endpoint model backing LRO: projects/676946291090/locations/us-central1/endpoints/5390144648939307008/operations/357529639112933376\n",
       "Endpoint model deployed. Resource name: projects/676946291090/locations/us-central1/endpoints/5390144648939307008\n"
      ]
     }
   ],
   "source": [
    "endpoint = # TODO 7: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serve predictions\n",
    "#### Prepare the input file with JSON formated instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Query the deployed model using `endpoint`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
     {
      "data": {
       "text/plain": [
        "Prediction(predictions=[1.0], deployed_model_id='7441226415950790656', model_version_id='1', model_resource_name='projects/676946291090/locations/us-central1/models/810790869438300160', explanations=None)"
       ]
      },
      "execution_count": 31,
      "metadata": {},
      "output_type": "execute_result"
     }
   ],
   "source": [
    "instance = [2841.0, 45.0, 0.0, 644.0, 282.0, 1376.0, 218.0, 237.0, 156.0, 1003.0, \"Commanche\", \"C4758\"]\n",
    "\n",
    "# TODO 8: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2022 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "    https://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m100",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m100"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
